Skip to main content

Running Demo Scenarios

Quick Demo: RL Training with Anomaly Detection​

This demo runs the SMOCS quick start guide with an autoencoder attached to show how multiple agents can pull from the same data stream.

This system spawns:

  • TD3 reinforcement learning agent learning Pendulum-v1
  • Autoencoder agent learning state space reconstruction

Prerequisites​

  • Docker and Docker Compose installed
  • SMOCS installed

Setup​

1. Configure environment:

cd SMOCS/orchestration
nano .env

2. Edit .env file:

COMPOSE_PROFILES=gymnasium,rl1,autoencoder1

We will now be utilizing the RL demo from the quickstart and add the autoencoder to learn how to reconstruct the state space. Please see the configuration of the autoencoder1 in the config.yaml to understand what variables are being used.

3. Launch demo:

docker compose up --build

What's Running​

Infrastructure:

  • Kafka broker on port 9092
  • InfluxDB on port 8086 (admin/admin123)

Active Services:

  • gymnasium-kafka-controller: Pendulum-v1 environment
  • rl-control-agent1: TD3 agent learning control policy
  • autoencoder-agent1: Anomaly detection on states
  • influxdb-consumer: Storing all metrics

Verify Operation​

Remember instead of using the CLI you can always view logs in Docker Desktop that allows for a UI!

Check logs for key indicators:

# Gymnasium state progression
docker compose logs gymnasium-kafka-controller | grep State

# Gymnasium publishing states
docker compose logs gymnasium-kafka-controller | grep Sent

# RL agent generating actions
docker compose logs rl-control-agent1 | grep actions

# Autoencoder detecting anomalies
docker compose logs autoencoder-agent1 | grep "Anomaly"

View metrics:

It may take approximately 5-10 minutes for statistics to begin populating the InfluxDB from initial start as the system is starting from cold. Please allow this time to pass before expecting logs in InfluxDB. Below is a list of expected behaviors.

Expected Behavior​

First 5 minutes:

  • Gymnasium publishes states every 100ms
  • RL agent uses random exploration
  • Autoencoder collects data (no training yet)
  • Episode rewards are low (around -1500)

After 5-10 minutes:

  • RL agent completes warmup (2500 samples)
  • Training begins, rewards improve
  • Autoencoder trains first model
  • Anomaly detection starts

After 30 minutes:

  • RL agent shows learning (rewards improve toward -200)
  • Autoencoder detects deviations from normal states
  • System reaches steady-state operation

Data Flow​

Gymnasium → state → RL Agent → action → Gymnasium
    ↓                                       ↓
  state                                   SARSA
    ↓                                       ↓
Autoencoder                          RL Training
    ↓                                       ↓
 anomalies                            better policy
    ↓                                       ↓
InfluxDB ←─────────────────────────────────┘

Stop Demo​

# Stop containers
docker compose down

# Clean slate (removes all data)
docker compose down -v

Troubleshooting​

No metrics in InfluxDB:

  • Check influxdb-consumer logs: docker compose logs influxdb-consumer
  • Verify Kafka topics exist: docker exec kafka-broker kafka-topics.sh --list --bootstrap-server localhost:9092
  • Give more time to the system to propogate the data from Kafka to InfluxDB!

RL agent not training:

  • Verify warmup completed: Look for "Training" in logs
  • Check buffer size: Logs show "buffer size: X"
  • Ensure Gymnasium is publishing: Check gymnasium logs

Autoencoder not training:

  • Verify data accumulation: Logs show "Database contains X samples"
  • Check minimum threshold: Needs 100 samples (configurable)
  • Review preprocessing errors: Look for "pipeline failed"